
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address COMMISSIONER FOR PATENTS 
P.O.Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



09/716,445 



9629 



11/21/2000 



Hitoshi Okamoto 



046601-5077 



8225 



7590 07/15/2004 

MORGAN LEWIS & BOCKIUS LLP 
1111 PENNSYLVANIA AVENUE NW 
WASHINGTON, DC 20004 



EXAMINER 



SUKHAPHADHANA, CHRISTOPHER T 

T 



ART UNIT 



PAPER NUMBER 



2625 

DATE MAILED: 07/15/2004 



~7 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



• 

Office Action Summary 


Application No. 

09/716,445 


Applicant(s) 

OKAMOTO, HITOSHI 


Examiner 

Christopher T. Sukhaphadhana 


Art Unit 

2625 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 04 June 2004 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1 935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) [3 Claim(s) 1-22 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-22 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 21 November 2000 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) K Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

aM All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) [3 Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 

Paper No(s)/Mail Date . 



4) O Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) CI Notice of Informal Patent Application (PTO-1 52) 

6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 7 



Application/Control Number: 09/716,445 Page 2 

Art Unit: 2625 

DETAILED ACTION 
Response to Amendment 
1. The Amendment filed 4 June 2004 has been entered in full. 



Claim Objections 

2. Claims 1, 2, 6, 17, 18, and 21 are objected to because of the following informalities: In 
regards to claims 1 and 21, "the image data pieces" on line 3 of the claim lacks proper 
antecedent basis. Consider restoring to —the image data piece—. In regards to claim 2, consider 
replacing "the image data" on line 4 of the claim with —the image data piece--. Limitation 1 of 
claim 1 associates the characteristic values with "an image data piece" and not the broad term 
"image data". In regards to claim 6, consider restoring "fuzz theory" to —fuzzy theory—. In 
regards to claims 17 and 18, consider removing the redundant phrase "having been executed" 
from the end of the claim. Appropriate correction is required. 



Response to Arguments 

3. Applicant's arguments regarding the Yaung and Desai rejections under 35 USC 103 on 
pages 10-12 of the Amendment filed 4 June 2004 have been fully considered but they are not 
persuasive. 

4. Applicant argues in substance that: 

5. Yaung and Desai does not teach or suggest "characteristic value extraction unit that 
extracts n types of characteristic value vectors". Specifically, Yaung extracts only four (not n) 
image search features, and Yaung's similarity matrix is not vector data. 
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6. First, Examiner agrees that Yaung' s similarity matrix is not vector data. However, the 
four image search features mentioned in Applicant's argument (average color, histogram color, 
positional color, texture) would constitute vector data. 

7. Second, Examiner concedes that Yaung probably does not cover the full range of n that 
Applicant intends to cover. However, Yaung still anticipates the limitation for at least n = 4. 
Consider also col 2, line 43, in light of col 2, line 18. 

8. Therefore, Yaung and Desai teach or suggest "characteristic value extraction unit that 
extracts n types of characteristic value vectors" and the remaining limitations of claim 1 to the 
extent required under 35 USC 103. 

9. Applicant's arguments regarding the Craver rejections under 35 USC 103 on pages 12-13 
of the Amendment filed 4 June 2004 have been fully considered but they are not persuasive. 

10. Applicant argues in substance that: 

1 1 . Craver does not teach a sorting register as claimed. Furthermore, Craver does not provide 
motivation for combining a sorting register unit. 

12. The section the Examiner references regarding a sorting register includes: "Additionally, 
in one embodiment, images further away from the center, or target image, are displayed with a 
smaller size to denote greater dissimilarity from those images closer to the center." (col 8, line 
39). This statement directly implies a sorting register structure to place the images in an order of 
similarity/dissimilarity from a center image. Therefore, Craver teaches or suggests a sorting 
register as claimed. See also the display structures in Fig 4 and 5. 

13. The motivation for combining Craver' s sorting register unit was presented in paragraph 
29 of the previous Office Action: "It would have been obvious to one of ordinary skill in the art 
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at the time of the invention to combine the first and second embodiments with the third 
embodiment because it allows easy selection of the target image from any of the displayed 
images (col 8, line 59) and images further away from the target image is displayed with a smaller 
size to denote greater dissimilarity (col 8, line 40)" 

14. No further specific arguments were directed towards the patentability of the remaining 
claims. 

Claim Rejections - 35 USC §103 

15. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

16. Claims 1-3, 6-9, 11, 12, and 16-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Yaung (U.S. Patent 6,512,850 B2, newly cited, "Yaung") in combination with 
Desai et al (U.S. Patent 6,072,904, newly cited, "Desai"). 

17. In regards to claim 1, Yaung discloses an image data sorting device (Fig 1) comprising: a 
characteristic value extraction unit (ref no 308, Fig 3, and col 3, line 49) that extracts 
characteristic values of an image data piece from the image data piece, wherein extracted 
characteristic values (col 3, line 49) are expressed in Ij dimensional vector (j = 1, 2, n, here 
1<= Ij, 1 <= n) in which n types of characteristic values vectors are obtained (col 6, line 20); a 
relation evaluation unit (ref no 316, Fig 3, and col 4, line 58) that evaluates a mutual relation 
between the characteristic value vectors that the characteristic value extraction unit extracts from 
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plural image data pieces; and a clustering unit (ref no 3 18, Fig 3, and col 5, line 3) that executes 
clustering to events expressed by the characteristic value vectors of the plural image data pieces 
on the basis of an evaluation result of the relation acquired by the relation evaluation unit. 

Yaung does not expressly disclose a sorting register as claimed. However, Yaung does 
teach in col 5, line 21, that image objects corresponding to the image object identifiers in one or 
more clusters may then be easily retrieved and visually displayed on visual output device. Yaung 
does not expressly teach how the image objects in the clusters are displayed or their manner of 
presentation. 

Desai teaches in col 5, line 65, a sorting register unit that automatically sorts the plural 
image data pieces on the basis of a result of clustering by the clustering unit. Specifically, Desai 
teaches a sorted list wherein the identifiers of the images are placed in the list in order of 
ascending distance from the target histogram vector. This ordered list is returned for presentation 
of the images to the user, typically by displaying the first ten closest pictures, then the next 10, 
etc. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to incorporate Desai 's sorting register unit into Yaung 5 s image data sorting device because it 
would present the user with images in a relevant order. These two teachings are compatible in 
the sense that Desai (col 5, line 63) uses a distance between target histogram vector and data base 
image histogram vectors as a measure of relevance, while Yaung (col 4, line 67) uses a similarity 
value as a measure of relevance. 
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Furthermore, the four image search features mentioned in Applicant's argument (average 
color, histogram color, positional color, texture) would constitute vector data. Yaung anticipates 
the limitation for at least n = 4 

18. In regards to claim 2, Yaung further discloses the device wherein: the characteristic 
extraction unit extracts the n types of characteristic value vectors as the characteristic value of 
the image data (col 3, line 49 and col 2, line 18); and the relation evaluation unit, synthesizing 
(col 4, line 57) the n types of characteristic value vectors as to each of the n types of the 
characteristic value vectors that the characteristic value extraction unit extracts from the plural 
image data pieces, evaluates the relation (col 4, line 64) between the image data pieces based on 
synthesis result. 

19. In regards to claim 3, Yaung further discloses in col 5, line 3, and Fig 5, "cluster J", the 
device wherein the clustering unit executes clustering by means of a non-hierarchical method 
based on a cluster number. 

20. In regards to claim 6, Yaung further discloses in col 5, line 9, "threshold criteria data" the 
clustering unit executing clustering by means of a crisp technique that does not employ the fuzzy 
theory. 

21 . In regards to claim 7, Yaung further discloses in Fig 5, "cluster J", the clustering unit 
including a cluster number specifying unit that specifies a cluster number when executing 
clustering. 

22. In regards to claim 8, Yaung further discloses the device further comprising: an image 
storage unit (ref no 106, Fig 1) that stores plural image data pieces; and an image management 



Application/Control Number: 09/716,445 Page 7 

Art Unit: 2625 

unit (ref no 202, Fig 2) that manages the plural image data pieces stored in the image storage unit 
on the basis of a sorting result by the sorting register unit. 

23. In regards to claim 9, Yaung further discloses in col 4, lines 1 5-26, the image 
management unit managing the plural image data pieces stored in the image storage unit on the 
basis of the sorting result by the sorting register unit and attribute information relating to the 
image data pieces, which are given to the plural image data pieces. 

24. In regards to claim 11, Yaung further discloses in col 12, line 7, the clustering unit and 
the sorting register unit executing clustering and sorting of the image data pieces stored in the 
image storage unit each time a specific period of time passes. 

25. In regards to claim 12, Yaung further discloses in col 12, line 1, the device wherein when 
there is a new image data piece to be stored in the image storage unit after clustering and sorting 
of the image data pieces stored in the image storage unit, the clustering unit and sorting register 
unit execute sorting of the new image data piece in such a manner that the new image data piece 
belongs to a cluster having the highest relation among existing clusters relating to the image data 
pieces stored in the image storage unit. 

26. In regards to claim 16, Yaung further discloses in col 12, line 4, the device wherein when 
a specific time passes after sorting by the sorting register unit, the clustering unit and the sorting 
register unit destroy the existing sorting and execute clustering and sorting of all of the image 
data pieces stored in the image storage unit. 

27. In regards to claim 17, Yaung further discloses in col 6, line 9, the device further 
comprising an image output unit that outputs image data pieces located near the center of each 
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cluster, upon completion of clustering by the clustering unit and sorting by the sorting register 
unit. 

28. In regards to claim 18, Yaung further discloses in col 6, line 9, the device further 
comprising an image output unit that outputs image data pieces having a high relation with 
image data pieces located near the center of each cluster, upon completion of clustering by the 
clustering unit and sorting by the sorting register unit. 

29. In regards to claim 19, Yaung further discloses in col 6, line 9, the image output unit 
outputs only the image data pieces relating to a designated cluster. 

30. In regards to claim 20, Yaung further discloses in col 11, line 67, the device wherein 
after clustering by the clustering unit and sorting by the sorting register unit have been executed, 
when clustering by the clustering unit and sorting by the sorting register unit are executed again, 
the image output unit again outputs image data pieces after being sorted again. 

31. In regards to claim 21, all the elements set forth in this claim have been addressed in the 
argument of claim 1 . 

32. In regards to claim 22, all the elements set forth in this claim have been addressed in the 
argument of claim 2. 

33. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yaung (U.S. 
Patent 6,512,850 B2, cited above, "Yaung") and Desai et al (U.S. Patent 6,072,904, cited above, 
"Desai") as applied to claim 1 above, in further combination with Lim (U.S. Patent 6,574,378 
Bl, newly cited, "Lim"). 
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34. In regards to claim 5, Yaung and Desai do not expressly disclose the clustering unit 
executing clustering by means of a fuzzy technique that employs a fuzzy theory. 

Specifically, Yaung discloses the clustering means using threshold criteria data (col 5, 
line 18). 

Lim teaches a clustering unit (col 7, lines 37-55) executing clustering by means of a 
fuzzy technique that employs the fuzzy theory. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to incorporate Lim's clustering unit in place of Yaung's clustering unit because fuzzy clustering 
is unsupervised (Lim, col 7, line 4) and Lim's fuzzy algorithm is the fuzzy equivalent of a 
classical "hard" clustering algorithm (col 7, line 38). 

35. Claims 10 and 15 are rejected under 35 U.S. C. 103(a) as being unpatentable over Yaung 
(U.S. Patent 6,512,850 B2, cited above, "Yaung") and Desai et al (U.S. Patent 6,072,904, cited 
above, "Desai") as applied to claim 8 above, in further combination with Hutcheson et al (U.S. 
Patent 5,161,204, newly cited, "Hutcheson"). 

36. In regards to claim 10, Yaung and Desai do not expressly disclose the device wherein 
when the image storage unit stores a specific number of image data pieces, the clustering unit 
and the sorting register unit execute clustering and sorting of the image data pieces stored in the 
image storage unit. 

Hutcheson teaches in col 17, lines 23-27, a device wherein when the image storage unit 
stores a specific number of image data pieces, the clustering unit and the sorting register unit 
execute clustering and sorting of the image data pieces stored in the image storage unit. 
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It would have been obvious to one of ordinary skill in the art at the time of the invention 
to incorporate the teachings of Hutcheson into the device of Yaung and Desai because it provides 
sufficiently reliable addressing of element clusters over large databases (Hutcheson, col 17, line 
41). 

37. In regards to claim 15, Yaung and Desai do not expressly disclose the device wherein 
when more than a specific number of image data pieces are added in the image storage unit after 
sorting by the sorting register unit, the clustering unit and the sorting register unit destroy the 
existing sorting and execute clustering and sorting of all the image data pieces stored in the 
image storage unit. 

Hutcheson teaches in col 17, lines 23-27, the device wherein when more than a specific 
number of image data pieces are added in the image storage unit after sorting by the sorting 
register unit, the clustering unit and the sorting register unit destroy the existing sorting and 
execute clustering and sorting of all the image data pieces stored in the image storage unit. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to incorporate the teachings of Hutcheson into the device of Yaung and Desai because it provides 
sufficiently reliable addressing of element clusters over large databases (Hutcheson, col 17, line 
41). 

38. Claim 13 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yaung (U.S. 
Patent 6,512,850 B2, cited above, "Yaung") and Desai et al (U.S. Patent 6,072,904, cited above, 
"Desai") as applied to claim 8 above, in further combination with Sato et al (JP 1 1-136573, 
newly cited, "Sato"). 
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39. In regards to claim 13, Yaung and Desai do not expressly disclose the clustering unit and 
sorting register unit executing sorting of the new image data piece on the basis of the relation 
with the characteristic value acquired from the center of gravity of the existing clusters relating 
to the image data pieces stored in the image storage unit. 

Sato teaches in paragraph 0026 the the clustering unit and sorting register unit executing 
sorting of the new image data piece on the basis of the relation with the characteristic value 
acquired from the center of gravity of the existing clusters relating to the image data pieces 
stored in the image storage unit. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to incorporate Sato's teachings into Yaung and Desai's device because it enables the addition of 
information at one time by collecting the whole classification and enabling it to add information 
by its classification (Sato, paragraph 0014). 

40. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yaung (U.S. 
Patent 6,512,850 B2, cited above, "Yaung") and Desai et al (U.S. Patent 6,072,904, cited above, 
"Desai") as applied to claim 8 above, in further combination with Craver et al (U.S. Patent 
6,233,367 Bl, newly cited, "Craver"). 

In regards to claim 14, Yaung and Desai do not expressly disclose the clustering unit and 
the sorting register unit executing sorting of the new image data piece on the basis of a center 
value of the relations between the characteristic value of the new image data piece and the 
characteristic values of the image data pieces each stored in the image storage unit. 
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Craver teaches in col 7, line 67, the clustering unit and the sorting register unit executing 
sorting of the new image data piece on the basis of a center value of the relations between the 
characteristic value of the new image data piece and the characteristic values of the image data 
pieces each stored in the image storage unit. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to incorporate Craver' s teachings into Yaung and Desai's device because the target image can be 
compared with an entire group (col 8, line 2), thereby saving computational time. 

41. Claims 1, 2, 4, 21, and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Craver et al (U.S. Patent 6,233,367 Bl, cited above, "Craver"). 

42. In regards to claim 1, Craver discloses in one embodiment an image sorting device (Fig 
6) comprising: a characteristic value extraction unit (col 7, lines 1-7) that extracts a characteristic 
value of an image data piece from the image data piece, wherein extracted characteristic values 
are expressed in Ij dimensional vector (j = 1, 2, n, here l<=Ij, 1<= n) in which n types of 
characteristic value vectors are obtained (col 7, lines 3-7 and 14-16); and a relation evaluation 
unit (col 7, lines 8-13) that evaluates a mutual relation between the characteristic value vectors 
that the characteristic value extraction unit extracts from plural image data pieces. 

Craver discloses in another embodiment a clustering unit (col 7, line 63) that executes 
clustering to events expressed by the characteristic value vectors of the plural image data pieces 
on the basis of an evaluation result of the relation acquired by the relation evaluation unit. 
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Craver discloses in a third embodiment a sorting register unit (col 8, lines 37-45) that 
automatically sorts the plural image data pieces on the basis of a result of clustering by the 
clustering unit. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to combine the first Craver embodiments with the second Craver embodiment because like 
images would be more likely to be placed next to each other (col 7, line 54). It would have been 
obvious to one of ordinary skill in the art at the time of the invention to combine the first and 
second embodiments with the third embodiment because it allows easy selection of the target 
image from any of the displayed images (col 8, line 59) and images further away from the target 
image is displayed with a smaller size to denote greater dissimilarity (col 8, line 40). 

The section the Examiner references regarding a sorting register includes: "Additionally, 
in one embodiment, images further away from the center, or target image, are displayed with a 
smaller size to denote greater dissimilarity from those images closer to the center." (col 8, line 
39). This statement directly implies a sorting register structure to place the images in an order of 
similarity/dissimilarity from a center image. Therefore, Craver teaches or suggests a sorting 
register as claimed. See also the display structures in Fig 4 and 5. 

43. In regards to claim 2, Craver further discloses the device wherein: the characteristic 
extraction unit extracts the n types of characteristic value vectors (col 7, lines 3-7 and 14-16) as 
the characteristic value of the image data; and the relation evaluation unit (col 7, lines 8-13), 
synthesizing the n types of the characteristic value vectors to each of the n types of characteristic 
value vectors that the characteristic value extraction unit extracts from the plural image data 
pieces, evaluates the relation between the image data pieces based on synthesis result. 
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44. In regards to claim 4, Craver further discloses in col 7, line 50, the device wherein the 
clustering unit executes clustering by means of a hierarchical method that joins clusters on the 
basis of the relation between the clusters. 

45. In regards to claim 21 3 all the elements set forth in this claim have been addressed in the 
argument of claim 1 . 

46. In regards to claim 22, all the elements set forth in this claim have been addressed in the 
argument of claim 2. 



Conclusion 

47. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Japanese KOKAI JP-1 1136573 discloses a video information adding method which 
sorts the representative images according to feature value vectors. 

48. Applicant's amendment necessitated the new ground(s) of objection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1. 136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 



Application/Control Number: 09/716,445 



Page 15 



Art Unit: 2625 

however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

49. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Christopher T. Sukhaphadhana whose telephone number is (703) 
306-4148. The examiner can normally be reached on 9a-5p M-F. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh M. Mehta can be reached on (703) 308-5246. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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